Evaluating prediction systems in software project estimation

This is the Pre-print version of the Article - Copyright @ 2012 ElsevierContext: Software engineering has a problem in that when we empirically evaluate competing prediction systems we obtain conflicting results. Objective: To reduce the inconsistency amongst validation study results and provide a more formal foundation to interpret results with a particular focus on continuous prediction systems. Method: A new framework is proposed for evaluating competing prediction systems based upon (1) an unbiased statistic, Standardised Accuracy, (2) testing the result likelihood relative to the baseline technique of random ‘predictions’, that is guessing, and (3) calculation of effect sizes. Results: Previously published empirical evaluations of prediction systems are re-examined and the original conclusions shown to be unsafe. Additionally, even the strongest results are shown to have no more than a medium effect size relative to random guessing. Conclusions: Biased accuracy statistics such as MMRE are deprecated. By contrast this new empirical validation framework leads to meaningful results. Such steps will assist in performing future meta-analyses and in providing more robust and usable recommendations to practitioners.Martin Shepperd was supported by the UK Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/H050329

Integrate the GM(1,1) and Verhulst models to predict software stage effort

This is the author's accepted manuscript. The final published article is available from the link below. Copyright @ 2009 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.Software effort prediction clearly plays a crucial role in software project management. In keeping with more dynamic approaches to software development, it is not sufficient to only predict the whole-project effort at an early stage. Rather, the project manager must also dynamically predict the effort of different stages or activities during the software development process. This can assist the project manager to reestimate effort and adjust the project plan, thus avoiding effort or schedule overruns. This paper presents a method for software physical time stage-effort prediction based on grey models GM(1,1) and Verhulst. This method establishes models dynamically according to particular types of stage-effort sequences, and can adapt to particular development methodologies automatically by using a novel grey feedback mechanism. We evaluate the proposed method with a large-scale real-world software engineering dataset, and compare it with the linear regression method and the Kalman filter method, revealing that accuracy has been improved by at least 28% and 50%, respectively. The results indicate that the method can be effective and has considerable potential. We believe that stage predictions could be a useful complement to whole-project effort prediction methods.National Natural Science Foundation of China and the Hi-Tech Research and Development Program of Chin

What are Hybrid Development Methods Made Of?

Regardless of company size or industry sector, a majority of project teams and companies use customized processes that combine different development methods-so-called hybrid development methods. Even though such hybrid development methods are highly individualized, a common understanding of how to systematically construct synergetic practices is missing. Based on 1,467 data points from a large-scale online survey among practitioners, we study the current state of practice in process use to answer the question: What are hybrid development methods made of? Our findings reveal that only eight methods and few practices build the core of modern software development. This small set allows for statistically constructing hybrid development methods

What are Hybrid Development Methods Made Of? An Evidence-Based Characterization

Among the multitude of software development processes available, hardly any is used by the book. Regardless of company size or industry sector, a majority of project teams and companies use customized processes that combine different development methods— so-called hybrid development methods. Even though such hybrid development methods are highly individualized, a common understanding of how to systematically construct synergetic practices is missing. In this paper, we make a first step towards devising such guidelines. Grounded in 1,467 data points from a large-scale online survey among practitioners, we study the current state of practice in process use to answer the question: What are hybrid development methods made of? Our findings reveal that only eight methods and few practices build the core of modern software development. This small set allows for statistically constructing hybrid development methods. Using an 85% agreement level in the participants’ selections, we provide two examples illustrating how hybrid development methods are characterized by the practices they are made of. Our evidence-based analysis approach lays the foundation for devising hybrid development methods

A simulation framework to support software project (re)planning

Planning and replanning software projects involves selecting activities according to organisational policies, project goals and contexts, deciding how to effect the activities, and dealing with uncertainty in activity outputs. There is at the present time no general model to support project managers with all of these tasks. The contributions of this paper are to propose a set of properties that are desirable in a model for (re)planning and to create a framework based on these properties. The purpose of the framework is to support the modelling and simulation of (re)planning during software projects. Key aspects of the framework are a focus on project objectives as drivers of activity selection, and activity prediction that supports uncertainty and that may be based on previous activity data, expert opinion or experimental evidence. We present a 'proof-of-concept' case study to illustrate how the framework can be applied to support planning

Managing Requirements Change the Informal Way: When Saying 'No' is Not an Option

© 2016 IEEE. Software has always been considered as malleable. Changes to software requirements are inevitable during the development process. Despite many software engineering advances over several decades, requirements changes are a source of project risk, particularly when businesses and technologies are evolving rapidly. Although effectively managing requirements changes is a critical aspect of software engineering, conceptions of requirements change in the literature and approaches to their management in practice still seem rudimentary. The overall goal of this study is to better understand the process of requirements change management. We present findings from an exploratory case study of requirements change management in a globally distributed setting. In this context we noted a contrast with the traditional models of requirements change. In theory, change control policies and formal processes are considered as a natural strategy to deal with requirements changes. Yet we observed that "informal requirements changes" (InfRc) were pervasive and unavoidable. Our results reveal an equally 'natural' informal change management process that is required to handle InfRc in parallel. We present a novel model of requirements change which, we argue, better represents the phenomenon and more realistically incorporates both the informal and formal types of change

The viability of fuzzy logic modeling in software development effort estimation: Opinions and expectations of project managers

There is a growing body of evidence to suggest that significant benefits may be gained from augmenting current approaches to software development effort estimation, and indeed other project management activities, with models developed using fuzzy logic and other soft computing methods. The tasks undertaken by project managers early in a development process would appear to be particularly amenable to such a strategy, particularly if fuzzy logic models are used in a complementary manner with other algorithmic approaches, thus providing a range of predictions as opposed to a single point value. As well as providing a more intuitively acceptable set of estimates, this would help to reduce or remove the unwarranted level of certainty associated with a point estimate. Furthermore, such an approach would enable organizations to "store" their project management knowledge, making them less susceptible to employee resignations and the like. If fuzzy logic modeling is to be implemented in industry, however, managers must first believe it to be a realistic and workable option. This issue is addressed here by considering two related questions: one, what expectations do project managers have in relation to effort estimation? And two, what is their opinion of the methods that might be useful in this regard? This is followed by a discussion of the results of two surveys of project managers aimed at deriving membership functions using polling methods, the first using an interval declaration approach and the second using votes on fixed points. It is concluded that there is indeed support in the software engineering practitioner community for the use of methods based on the principles of fuzzy logic modeling

Using historical data in stochastic estimation of software project duration

This paper presents a framework for the representation of uncertainty in the estimates used to predict the duration of software design projects. The modelling framework utilises Monte Carlo simulation to compute the propagation of uncertainty in estimates towards the total project uncertainty and therefore gives a project manager the means to make informed decisions throughout the project life. The framework also provides a mechanism for accumulating project knowledge through the use of a historical database, allowing effort estimates to be informed by, or indeed upon, the outcome of previous projects

Analysis of ALTAIR 1998 Meteor Radar Data

We describe a new analysis of a set of 32 UHF meteor radar traces recorded with the 422 MHz ALTAIR radar facility in November 1998. Emphasis is on the velocity measurements, and on inferences that can be drawn from them regarding the meteor masses and mass densities. We find that the velocity vs altitude data can be fitted as quadratic functions of the path integrals of the atmospheric densities vs distance, and deceleration rates derived from those fits all show the expected behavior of increasing with decreasing altitude. We also describe a computer model of the coupled processes of collisional heating, radiative cooling, evaporative cooling and ablation, and deceleration - for meteors composed of defined mixtures of mineral constituents. For each of the cases in the data set we ran the model starting with the measured initial velocity and trajectory inclination, and with various trial values of the quantity mPs 2 (the initial mass times the mass density squared), and then compared the computed deceleration vs altitude curves vs the measured ones. In this way we arrived at the best-fit values of the mPs 2 for each of the measured meteor traces. Then further, assuming various trial values of the density Ps, we compared the computed mass vs altitude curves with similar curves for the same set of meteors determined previously from the measured radar cross sections and an electrostatic scattering model. In this way we arrived at estimates of the best-fit mass densities Ps for each of the cases. Keywords meteor ALTAIR radar analysis 1 Introduction This paper describes a new analysis of a set of 422 MHz meteor scatter radar data recorded with the ALTAIR High-Power-Large-Aperture radar facility at Kwajalein Atoll on 18 November 1998. The exceptional accuracy/precision of the ALTAIR tracking data allow us to determine quite accurate meteor trajectories, velocities and deceleration rates. The measurements and velocity/deceleration data analysis are described in Sections II and III. The main point of this paper is to use these deceleration rate data, together with results from a computer model, to determine values of the quantities mPs 2 (the meteor mass times its material density squared); and further, by combining these m s 2 values with meteor mass estimates for the same set of meteors determined separately from measured radar scatterin